Test build #72979 has finished for PR 16949 at commit a3d2abb.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

jenkins crushed. retest this please.

Test build #72986 has started for PR 16949 at commit a3d2abb.

cc @srowen

terminated by signal 9. retest this please.

Test build #72989 has finished for PR 16949 at commit a3d2abb.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

It's a simple change, but I wonder if this is that important to add?
I always have a worry in the back of my mind that this becomes a security hole, as it's a way to look through the complete environment of a bunch of jobs.

@srowen good question！IMHO，we should add this API:

• provide complete API, the same as users see in webui
• if this is a security issue, we should address it in other ways
• existing API also has security issue as you said. Maybe, we need some authorization check or something else, just you said security issue.

Any suggestion is appreciated!

@uncleGen As followup I'd recommend opening a JIRA to research and address the potential security flaws in the env tab. I know we've already merged some fixes to hide passwords in the env tab, but we should make sure there aren't more issues.

re: security, this change doesn't really change anything; all this information is already available in the UI. Note it doesn't expose the system env (as in environment variables), although people may still expose things through system properties and spark configuration. There's some code to redact sensitive info and even a config to control what gets redacted, but ultimately, the solution to security is auth + ssl. As Spark gets more and more features, data may start leaking through other places (e.g. SQL query plans which are shown in the UI).

As for the change, I kinda implemented it as part of my SHS project, and I went with a slightly different API:
vanzin@d127f15

Mainly I broke the "JVM properties" into a separate type; I kinda prefer that instead of "magic keys", especially when these keys contain spaces and thus don't map well to code.

@vanzin I opened a jira (https://issues.apache.org/jira/browse/SPARK-19642) to research and address the potential security flaws. Do you mind if I continue this pr?

Sure, this PR is fine, I'd just prefer some minor API adjustments to bring it closer to the code I linked above.

Thanks @uncleGen and after seeing his code I agree with @vanzin

@vanzin @ajbozarth sure, I will do some update based on the linked code.

Test build #73145 has finished for PR 16949 at commit ad570cf.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

LGTM

cc @srowen and @vanzin also.

Test build #73309 has finished for PR 16949 at commit 6094743.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

Merging to master.